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ABSTRACT 

Differences in judged relevance of Educational 
Resources Information Center (ERIC) abstracts identified by two 
different information systems were investigated. . A. national 
semi-private educational service agency which utilizes mechanical 
searching of the research literature was one information system, 
while the other was a regional, Federally-funded agency employing 
manual searching. Twenty clients reported they found substantially 
more ERIC abstracts identified by a manual method to be relevant to 
their needs than abstracts located by a mechanical searching system 
using the same set of ERIC descriptors. (Author) 
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The number of information systems and centers In educational settings 

! 

Is increasing nationally and more are contemplated. Many of these information 
retrieval systems are automated and are intended to provide information to the 

i 

user/educator quickly and efficiently. V/hether or not these systems or centers 

t 

are providing relevant information to requestors is not clear. To provide 
preliminary data which would begin to serve as a basis for rational decisions 
continues to be a problem in education. 

| i 

To improve research information service, and to provide a foundation 

i I 

for valid judgments about the efficacy of providing relevant information via 
a tomated retrieval systems, a modest experimental pilot research study was 
undertaken. It was decided to conduct a comparative study of two different 
-information retrieval systems using Educational Resources Information Center 
materials (E.R.I.C.). A national semi-private research information agency 

I 

(N.I.A.) which utilizes machine searching of the research literature was 
compared with the Research and Information Services for Education (R.I.S.E.), a 
regional research information agency employing manual searching of the research 
literature. 

R.t.S.E., an agency funded under Title 1 1 1 of the Elementary and 

« t 

Secondary Education Act of 1965, is an educational research agency functioning 
at the local -intermediate-regional level. One of its more visible activities is 
to provide research information on request to educators. During the first two 
years of existence, over 800 , requests for information were handled. In an 
effort to improve its information service, R.I.S.E. has continually reviewed 
and evaluated both the process and product of its efforts. 
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A short evaluation form asking for reactions to R.I.S.E. service and 
to the product was included with each packet of information that was sent out. 
Most clients returned the form, which, in general, showed a highly favorable 
■response. Supplementing this effort were other evaluation procedures. These 
Incl uded 'on-si te evaluations by outside experts as well as mandated evaluations 
by State evaluation groups. In addition, a random sampling of clients using 
typical interview techniques was conducted by the R.I.S.E. staff. 

N.I.A. , supported by private foundation funds, small fees from users, 

♦ 

? and Federal monies, is an educational service agency supplying research inform- 
ation to users drawn from throughout the county. Its information service 

employs optical coincidence scanning to serve the increasing number of their 
clients. 

Purpose 

The ..sped f ic purpose of the comparison is to obtain some preliminary 
data to determine the difference in relevance of information received, as 
judged by the user. "Relevance" was defined as the relationship between a 
document and the user's information requirement.^ While relevance judgments 
are complex processes, ^ it was decided to focus on this criterion because a 

demand literature search is the basic premise on which R.I.S.E. continues to 

\ 

serve its clients. Thus, "relevance" has more importance to R.I.S.E. than the 
issues of "speed," "cost," or "accuracy," which are considered secondary for 
purposes of this study. 

A true experimental study, employing inferential statistical approaches, 
was not attempted. 

% 

Methodology 

In a regular search of the E.R.I.C. collection, it is customary for 
R.I.S.E. to send to the client abstracts along with either excerpts of 
pertinent sections of a report or the complete report itself if not too lengthy. 

ERJC 



It was decided, for purposes of the study, that only abstracts would be 
forwarded to the client by R.I.S.E. Since the N.I.A. sends only the accession 

number for any E.R.I.C. abstract, and not the abstract Itself*, to the requestor, 

♦ * 

a decision by R.I.S.t. to make a selection of excerpts from printed articles 

* 

would have Invalidated the study. The alternative of sending the complete 
printout for each item was ruled out on a financial and practical basis, since 

many E.R.I.C. documents run to several hundred pages. 

* 

It Is. known that there is a tendency to judge items more relevant if 
less is known about the subject and the person does not have sufficient evidence 
to decide between relevance and non-relevance .* 1 Hence, it was known there would 
be instances where on the basis of an abstract alone, a client might make one 
decision regarding relevancy, but then change this decision after reading the 
complete report. Despite this, it was decided that providing only abstracts - 
while not an Ideal method - was the most feasible course in a pilot study and 
further, that if a client erred because the abstract did not provide sufficient 
information, he would err in favor of both R.I.S.E. and N.I.A. 

- Four possible responses were decided upon: "relevant," "not relevant," 

"don't know but possibly relevant," or "don't know but possibly not relevant." 

■*% . 

The last two were selected because it was probable there would be a good number 
of items about which the client could not make a clear decision between 
"relevant" and "not relevant." It was deemed inappropriate to include only 
one additional possibility - "don't know" - since it is logical to expect certain 
instances where a client cannot decide between "relevant" and "don't Know" or 
between "not relevant" and "don't know." In such cases the client might check 

Jt relevant" Instead of "don't know," not on the basis of the content of the 

/ 

abstract, but on the oasis of past experiences with tests or evaluations, or of 
speculations about how the evaluation might be set up. It was believed that 

i , « 

two choices within the "don't know" category might help solve these problems. 



The possibility of providing a fifth choice - ’’don't know” - in addition 

| . 

to "don't know but possibly relevant" and "don't know but possibly not relevant" 
was considered. 5 it was ultimately decided that this was unnecessary since if 
* a client could not make any decision about an abstract, it probably should be 

4 

listed under "don't know but possibly not relevant," that is, both have the 
same meaning. Further, having a fifth choice would create a false value for 
choices three and four: without the fifth possibility, the client would have 
no qualms about checking "don't know but possibly relevant." If there were 
the "fifth choice, however, the client might assume that cho*ce three had "stricter 
criteria" than intended. 

Procedure 

In general, it was decided for purposes of the study, to accept the next 

20 literature search requests in the order received by R.I.S.E. Excluded were 

—certain., types... of requests so specialized that the ..topics would not likely be 

found in E.R.I.C. - for instance, patterns revealed by Rorschach tests; or 

because they were requests of a high priority nature, which necessitated unusual 
* 

and hence a typical processing" and' treatment ; or because they were requests 
. submitted by clients under special contract. In the last case, it was felt 

that these clients might give biased responses because of the fee charged them. 

• \ 

Following acceptance of a search topic for the study, a detailed state- 
ment of the request was sent to the N.I.A., including a list of descriptors 
arrived at between the negotiator and the client.^ At "the .beginning of the 
study, both agencies agreed that this procedure was mutually satisfactory and 
that no additional information on the question need be sought. 

R.I.S.E. literature searchers were informed of the client's request in 
the usual way; they could ask any questions of interpretation of the supervisor 
who mediated the request initially. The searchers were asked to complete the 
E.R.I.C. portion of the search first and then fill out a special worksheet on 



which to list accession numbers and so on, plus one extra copy of each applicable 

i 

i 

. abst. jet. Searchers were informed this was part of a special pilot study, but 

were told to perform the E.R.I.C. portion of the search in the normal way. 

i 

When the abstract accession numbers were received from the N.l.A. , these 
numbers were copied onto R.I.S.E. 's special worksheet. The searcher was then 

i 

asked to make one copy of each abstract listed. 

An evaluation form^ was prepared for the client; it listed in numerical 
order all abstracts retrieved by both information agencies. The numbers from 
each v/ere interdigi tated so that the client would have no way of knowing which 
abstracts came’ from which source. If both agencies had selected the same abstract 

« i 

only one entry was made on the evaluation form. The client also received one 

|: • 

copy of each of the abstracts listed and was asked to judge the relevance on a 

i I 

subjective basis, that is, how a particular abstract related to his specific 
question not how it might relate to his subject In general. The clients were 
also informed that this pilot study was part of a special project and would 
^■rnterfere'Tn"no way wi th their regular ongoing search, the results of which 
would be mailed as soon as his evaluation form was completed and returned to 
R.I.S.E. 

* y * 

Upon receipt of the evaluation form the data were transferred to a summary 
sheet as follows: abstracts retrieved from both sources were listed first, 

along with the subjective responses; next came a listing of abstracts retrieved 
only by N.l.A. ; the third group was. the R.I.S.E. abstracts. The summary of 
results was tabulated at the bottom of the worksheet. 

The pilot study was conducted from April to July 1969 and involved 20 
respondents.^ 

Results 

Table 1 presents data regarding the differences in relevance, as reported 
by users, between E.R.I.C. abstracts selected by R.I.S.E. and those selected 



Insert Table 1 about here 



An examination of the data reveals, first, that R.I.S.E. searchers located 

j 

almost twice as many abstracts as did N.I.A.: 118 or 68% of the total number 

j , 

of 173 abstracts on which clients expressed judgments for R.I.S.E., and 55 or 

i 

32% for N.I.A. Of the 118 abstracts identified by R.I.S.E., 56 or V/% were 

i 

i 

judged by users to be "relevant," and 26 or 22% as "not relevant." By 
contrast, of the 55 abstracts located by N.I.A., the clients checked 23 or 37% 
as "relevant," and 17 or 27% as "not relevant." 

t 

\ 

In terms of absolute figures, the comparison is even more striking. The 

■l! 

users judged 56 abstracts located by R.I.S.E. as being relevant to their needs 

; ! 

and only 26 as being not relevant, whereas they judged 23 abstracts identified 

I . . ' . 

by N.I.A. as relevant and 17 as being not relevant. In addition, the 20 clients 
reported 3*» abstracts identified by R.I.S.E. as being "possibly relevant," 
compared to only 11 abstracts similarly identified by N.I.A. Clearly, then, the 
data show that the users judged many more R. I .S. E. -'identified than N.I.A.- 
identified abstracts as being relevant. 

• "V ' 

From the viewpoint of individual cases, of 15 clients (four clients 
reported no relevant abstracts from either agency, and a fifth reported one 
relevant abstract from each agency), 11 reported more relevant abstracts 
located by R.I.S.E., while only k reported more relevant abstracts located by N. 



v Discussion 

This study is subject to the usual limitations of studies involving a 
small number of cases; the reader is cautioned about the dangers of generalizing 
beyond the attempted purpose and limited scope of the study. Also, even by 
taking the requests for information in the order received, thus mirroring the 
"typical situation," variables, such as geographical proximity to the R.I.S.E. 
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center, were not completely controlled for. On the other hand, interdig i tat ing 

i 

I 

the abstracts and providing the same set of descriptors for both agencies did 
provide some measure of control. 

Additional research is needed to confirm or deny these preliminary 
findings. For instance, a more statistically sophicated experimental study 
might be used as a follow-up of the limited pilot study. The response format 
could be designed so as to admit an examination by inferential statistical 
methods. Each response option could be weighed and a t^ test applied. Such 
refinements may yield greater precision. In the meantime, on the basis of the 
limited data collected and analyzed descriptively, R.i.S.E.- located (manually 
selected) abstracts were judged by users to be more relevant than N.I.A. (machine 
selected) abstracts. * 
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| Footnotes 

I 

‘ * .1 * 

1 ... 1 

The fictitious name "National Information Agency" was chosen for purposes of 
the study. 1 

^Donald King, "Design and Evaluation of information Systems." In Annual 
Review of Information Science and Technology , Volume III, Encyclopedia 
Britanica, Inc., 1968, p. 70 . 

i 

3 

Carlos Cuadra and Robert Katter, Special Libraries , "Implications of 
Relevance Research for Library Operations and Training" (Volume LIX, 

September 1968), p. 50*». 

Sbid . , p. 506. 

5 

There was a point of view expressed in the early planning stages which argued 
for a fifth "don't know" choice, which would accomodate the client who 
definitely and honestly did not know. The provision for this fifth choice, 

It was held, would provide greater leeway on the continuem of a closure type 
response format. This argument was rejected in an initial pilot study for 
-the reasons cited in ( the text. Perhaps a second research study should be 
made, employing a fifth choice. 

6 1 1 

Robert S. Taylor, College and Research Libraries "Question-Negotiation and 
Information Seeking in Libraries" (May 1968T] Mediating of all requests was 
done by a single full-time professional who was fully aware of the communication 
gaps" in the negotiation process. The average mediation, received over the 
telephone, took at least 20 minutes. 

7 

See Appendix A. 

8 

®0f_.the 20 clients, 6 were teachers, 6 were directors, 3 were principals, I 
was a research intern, I was an elementary school supervisor, I was an assistant 
superintendent, I was a chief school administrator, and I was a college 
professor. There were 10 males and 10 females. In view of the small number 
of subjects, it was deemed not appropriate to analyze the data by staff 
position and/or sex. v 
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, • • . ■••••■' • • APPENDIX A • ••• 

' * * ' « * * < . ... 

• • 

R. J.. r S;E.' SPECIAL E V A L U AT ION _ ERIC MATERIALS 

• I , , 1 ' * 

• ( * | * ^ | ^ ^ 

Name? . 

* * * i « 

H PROJECT TITLE ' ’ . 
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DATE 



INSTRUCTIONS: The attached abstracts were selected by two different retrieval systems. 

The purpose of this evaluation Is to provide a basis for judging the value of each 
system, as part of RISERS effort to Improve Its literature search services. 

Please review each abstract and indicate your evaluation of its relevance, based on 
pertinence to your specific problem or question . Since these materials were prepared 
especially for your request, you are the only^ one who can Judge their relevance. • 

These materials do not represent a completed literature search. Your project Is In 
preparation and will be sent to you upon. receipt of this form. . ’ • 



Check only one category for each abstract: 



Abstract 
Numbe r 

E D 

ED 

E D 

ED 

ED 

ED_ 

ED 



Relevant 



Not.. 

Relevant 



Don’t Know, 
But possibly 
Relevant 



Don’t Kno w. 
But possibly 
Not Relevant 



ED 

ED’ 

ED’ 

ED’ 

ED’ 

ED* 

ED’ 

ED’ 



ED 

ED* 

ED* 



D 



, * 



omments, if any: • ~ 

hank you for participating In this evaluation. We hope your efforts will be rewarded 
y improved RISE service in the future. Please return this form in the enclosed envelope 



y improved 

a: # • ... 

Supervisor of Dissemination * • 

Research and Information Services for Education 
M3 South Gulph Road 
King of Prussia, Pennsylvania IM06 



o 

ERIC 



tn ♦ 

•* 

i 

I 

i 



Abstract 

Differences in judged relevance of E.R.I.C. abstracts identified by 
two different information systems were investigated. A national semi-private 
educational service agency which utilizes mechanical searching of the research 

literature was one information system, while the other was a regional, 

’ * V * ' »:'• 

Federally-funded agency employing manual searching., 20 clients reported they 

HI • » • *«■*“** •■■•It. . « 

found substantially more E.R.I.C. abstracts identified by a manual method to 
be relevant to their needs than abstracts located by a mechanical searching 
system using the same set of E.R.I.C. descriptors. 






